跳转至

文章背景与核心概要

检索增强生成（RAG）系统对于将大语言模型（LLM）的响应建立在事实依据上至关重要。然而，当前的评估框架在面对从简单的寻事实查询到复杂的开放式解释等各种用户查询时，往往难以提供一致且细粒度的诊断。

本文介绍了 Q-CARE，这是一个与查询无关且完全无需参考答案的评估框架。Q-CARE 通过将查询分解为子查询、将答案分解为原子声明，并建立基于查询覆盖率和声明可验证性的统一评估原则，从而解决了上述挑战。

该框架为检索（C-Prec@k、C-nDCG@k）和生成（完整性、简洁性和可验证性）均提供了特定的指标。在包含八个数据集的人工标注基准上进行验证表明，与 RAGEval 和 RAGChecker 等现有框架相比，Q-CARE 表现出与人类判断更高的相关性。

Towards Query-Agnostic RAG Evaluation via Query Coverage and Claim Verifiability

arXiv: 2608.11238 [cs.AI]
Accepted to: COLM 2026
Authors: Jeonghwan Choi, Taewon Yun, Minjeong Ban, Gyeonghun Sun, Jae-Gil Lee, Hwanjun Song

arXiv: 2608.11238 [cs.AI]
Accepted to: COLM 2026
Authors: Jeonghwan Choi, Taewon Yun, Minjeong Ban, Gyeonghun Sun, Jae-Gil Lee, Hwanjun Song

Summary

Retrieval-Augmented Generation (RAG) systems are essential for grounding LLM responses in factual evidence. However, current evaluation frameworks often struggle to provide consistent, granular diagnostics across the wide variety of user queries—from simple fact-seeking to complex, open-ended explanations.

This paper introduces Q-CARE, a query-agnostic and fully reference-free evaluation framework. Q-CARE addresses these challenges by: * Decomposing queries into sub-queries. * Decomposing answers into atomic claims. * Establishing a unified evaluation principle based on query coverage and claim verifiability.

The framework provides specific metrics for both retrieval (C-Prec@k, C-nDCG@k) and generation (Completeness, Conciseness, and Verifiableness). Validated against a human-annotated benchmark across eight datasets, Q-CARE demonstrates a higher correlation with human judgment compared to existing frameworks like RAGEval and RAGChecker.

Summary

Retrieval-Augmented Generation (RAG) systems are essential for grounding LLM responses in factual evidence. However, current evaluation frameworks often struggle to provide consistent, granular diagnostics across the wide variety of user queries—from simple fact-seeking to complex, open-ended explanations.

This paper introduces Q-CARE, a query-agnostic and fully reference-free evaluation framework. Q-CARE addresses these challenges by: * Decomposing queries into sub-queries. * Decomposing answers into atomic claims. * Establishing a unified evaluation principle based on query coverage and claim verifiability.

The framework provides specific metrics for both retrieval (C-Prec@k, C-nDCG@k) and generation (Completeness, Conciseness, and Verifiableness). Validated against a human-annotated benchmark across eight datasets, Q-CARE demonstrates a higher correlation with human judgment compared to existing frameworks like RAGEval and RAGChecker.

Key Metrics

Q-CARE introduces a suite of metrics to provide a comprehensive view of RAG performance:

Key Metrics

Q-CARE introduces a suite of metrics to provide a comprehensive view of RAG performance:

Retriever Metrics

  • C-Prec@k: Coverage-aware precision at rank k.
  • C-nDCG@k: Coverage-aware normalized discounted cumulative gain at rank k.

Retriever Metrics

  • C-Prec@k: Coverage-aware precision at rank k.
  • C-nDCG@k: Coverage-aware normalized discounted cumulative gain at rank k.

Generator Metrics

  • Completeness: Measures how well the generated response covers the necessary information.
  • Conciseness: Evaluates the efficiency of the response.
  • Verifiableness: Assesses the accuracy and grounding of atomic claims.

Generator Metrics

  • Completeness: Measures how well the generated response covers the necessary information.
  • Conciseness: Evaluates the efficiency of the response.
  • Verifiableness: Assesses the accuracy and grounding of atomic claims.

Resources

Resources

Submission History

  • v1: 31 Jul 2026
  • v2: 21 Aug 2026 (Current version)

Submission History

  • v1: 31 Jul 2026
  • v2: 21 Aug 2026 (Current version)